Data processing method, apparatus and terminal

ABSTRACT

At a computing system comprising one or more processors and memory, the computing system is attached to a moving vehicle traveling along a road: collecting road data while the computing system is in motion, the road data comprising a two-dimensional streetscape image, a three-dimensional point cloud, and inertial navigation data; performing region segmentation on the two-dimensional streetscape image; extracting a ground region image based on a spatial position relation of the two-dimensional streetscape image, the three-dimensional point cloud, and the inertial navigation data; detecting at least one target road traffic marking in the ground region image; and controlling motion of the moving vehicle in accordance with three-dimensional coordinates of the at least one target road traffic marking that are calculated based on the spatial position relation of the two-dimensional streetscape image, the three-dimensional point cloud, and the inertial navigation data.

PRIORITY CLAIM AND RELATED APPLICATION

This application is a continuation-in-part application ofPCT/CN2016/112671, entitled “INFORMATION PROCESSING METHOD, DEVICE, ANDTERMINAL” filed on Dec. 28, 2016, which claims priority to ChinesePatent Application No. 201610031452.5, filed with the State IntellectualProperty Office of the People's Republic of China on Jan. 18, 2016, andentitled “INFORMATION PROCESSING METHOD, DEVICE, AND TERMINAL”, all ofwhich are incorporated herein by reference in their entirety.

TECHNICAL FIELD

This application relates to the field of computer technologies,specifically to the technical field of road data processing, and inparticular, to a street image processing method, a street imageprocessing apparatus, and a terminal.

BACKGROUND

In existing fields of car navigation, vehicle automatic driving, anddriving safety tips, the most basic and important technology is adetecting technology about road traffic markings, such as lane lines androad signs. Currently, mainstream technical solutions mainly include thefollowing two kinds. One solution is a detecting method based on changesof ground reflectivity: reflectivity of the road traffic markings isgenerally higher than reflectivity of other parts of a road surface.Therefore, a three-dimensional point cloud of a road scene space mayfirst be obtained, and then the road traffic markings are extractedmanually from the three-dimensional point cloud according to thisfeature. Because the solution directly detects and extracts the roadtraffic markings from the three-dimensional point cloud, a detectingresult may be affected if a point cloud of the three-dimensional pointcloud is sparse, occluded, missing, etc. Moreover, accuracy of thedetecting result may be directly affected if the ground reflectivityturns out to be weak or uneven. The other solution is a manual detectingmethod: obtain a two-dimensional streetscape image and thethree-dimensional point cloud of the road scene space, and use thetwo-dimensional streetscape image to apply colors to thethree-dimensional point cloud, so as to generate a colorful point cloud,and then manually extract the road traffic markings from the colorfulpoint cloud according to the two-dimensional streetscape image. Thesolution still directly extracts the road traffic markings from thethree-dimensional point cloud, so it may be possible that the detectingresult may be affected if the point cloud of the three-dimensional pointcloud is sparse, occluded, missing, etc. Furthermore, manual detectionis impractical and of low efficiency, easily causing mistakes.

SUMMARY

Embodiments of this application provide a data processing method,apparatus, and a terminal. By detecting road traffic markings in atwo-dimensional streetscape image, accuracy of a detecting result andpracticability of data processing may be improved.

A first aspect of the embodiments of this application provides a dataprocessing method, which may include:

at a computing system comprising one or more processors and memory, thecomputing system is attached to a moving vehicle traveling along a road:

-   -   collecting road data while the computing system is in motion,        the road data including a two-dimensional streetscape image, a        three-dimensional point cloud, and inertial navigation data;    -   performing region segmentation on the two-dimensional        streetscape image and extracting a ground region image based on        a spatial position relation of the two-dimensional streetscape        image, the three-dimensional point cloud, and the inertial        navigation data;    -   detecting at least one target road traffic marking in the ground        region image, the road traffic marking including a lane line        and/or a road sign;    -   controlling motion of the moving vehicle in accordance with        three-dimensional coordinates of the at least one target road        traffic marking that are calculated based on the spatial        position relation of the two-dimensional streetscape image, the        three-dimensional point cloud, and the inertial navigation data.

Preferably, performing region segmentation on the two-dimensionalstreetscape image and extracting a ground region image based on aspatial position relation of the two-dimensional streetscape image, thethree-dimensional point cloud, and the inertial navigation dataincludes:

-   -   establishing a coordinate system of an inertial navigation of a        three-dimensional space according to the inertial navigation        data;    -   registering the three-dimensional point cloud and the        two-dimensional streetscape image in the three-dimensional space        based on the coordinate system of the inertial navigation;    -   performing, by a registered three-dimensional point cloud, the        region segmentation on the two-dimensional streetscape image and        extracting the ground region image.

Preferably, the detecting at least one target road traffic marking inthe ground region image includes:

-   -   performing binary processing to the ground region image to        obtain a binary ground region image;    -   performing orthographic projection transformation to the binary        ground region image to generate a ground orthographic projection        image;    -   extracting at least one road traffic marking from the ground        orthographic projection image.

Preferably, the extracting at least one road traffic marking from theground orthographic projection image includes:

-   -   extracting at least one first road traffic marking from the        ground orthographic projection image and extracting an        interference image by using an image recognition method;    -   regarding the interference image as a binary mask to filter the        ground orthographic projection image;    -   extracting at least one second road traffic marking from the        filtered ground orthographic projection image by using the image        recognition method; and    -   combining the first road traffic marking and the second road        traffic marking that are extracted to obtain at least one target        road traffic marking.

Preferably, the calculating three-dimensional coordinates of the atleast one target road traffic marking based on the spatial positionrelation of the two-dimensional streetscape image, the three-dimensionalpoint cloud, and the inertial navigation data includes:

-   -   obtaining morphological data of the ground orthographic        projection image in the coordinate system of the inertial        navigation, where the morphological data includes position data        and posture data;    -   finding the three-dimensional coordinates of the at least one        target road traffic marking according to the morphological data.

A system comprising one or more processors and memory that includesinstructions which when executed by the one or more processors, causethe processors to perform the method set forth herein is also disclosed.A non-transitory computer-readable storage medium storing instructions,the instructions, when executed by the one or more processors, cause theprocessors to perform the method set forth herein is also disclosed.

By implementing the embodiments of this application, the followingbeneficial effects are obtained:

In the embodiments of this application, after collecting atwo-dimensional streetscape image, a three-dimensional point cloud, andinertial navigation data, performing region segmentation on thetwo-dimensional streetscape image and extracting a ground region imageby using the three-dimensional point cloud and the inertial navigationdata, and detecting at least one target road traffic marking in theground region image and inversing three-dimensional coordinates of thetarget road traffic marking. It may be avoided that a detecting resultis affected if a point cloud of the three-dimensional point cloud issparse, occluded, missing, or something, because the road trafficmarking is detected and obtained from the two-dimensional streetscapeimage rather than the three-dimensional point cloud. In addition, byaccurately performing region segmentation on, detecting, and extractingthe two-dimensional streetscape image by using the inertial navigationdata and the three-dimensional point cloud, accuracy of the detectingresult and practicability of data processing may be effectivelyimproved.

BRIEF DESCRIPTION OF THE DRAWINGS

To illustrate the technical solutions in the embodiments of the presentapplication or in the prior art more clearly, the following brieflyintroduces the accompanying drawings required for describing theembodiments or the prior art. Apparently, the accompanying drawings inthe following description show merely some embodiments of the presentapplication, and persons of ordinary skill in the art may still deriveother drawings from these accompanying drawings without creativeefforts.

FIG. 1 is a flowchart of a data processing method according to anembodiment of this application;

FIG. 2a is a schematic diagram of a two-dimensional streetscape imageaccording to an embodiment of this application;

FIG. 2b is a schematic diagram of registering a two-dimensionalstreetscape image and a three-dimensional point cloud according to anembodiment of this application;

FIG. 2c is a schematic diagram of a ground region image according to anembodiment of this application;

FIG. 2d is a schematic diagram of a binary ground region image accordingto an embodiment of this application;

FIG. 2e is a schematic diagram of a ground orthographic projection imageaccording to an embodiment of this application;

FIGS. 3A-3B, including subparts 3 a-3 f, are a schematic diagram of aprocess flow for removing interfering objects from the streetscape imageaccording to an embodiment of the application.

FIG. 4 is a flowchart of a street image processing method according toan embodiment of this application.

FIG. 5 is a flowchart of a street image processing method according toan embodiment of this application.

FIG. 6 is a schematic structural diagram of a terminal according to anembodiment of this application; and

FIG. 7 is a schematic structural diagram of a data processing apparatusaccording to an embodiment of this application.

DESCRIPTION OF EMBODIMENTS

The technical solutions in the embodiments of the present applicationare clearly described in the following with reference to theaccompanying drawings. Apparently, the embodiments to be described aremerely a part rather than all of the embodiments of the presentapplication. All other embodiments obtained by persons of ordinary skillin the art based on the embodiments of the present invention withoutcreative efforts shall fall within the protection scope of the presentinvention.

In the embodiments of this application, after collecting atwo-dimensional streetscape image, a three-dimensional point cloud, andinertial navigation data, performing region segmentation on thetwo-dimensional streetscape image and extracting a ground region imageby using the three-dimensional point cloud and the inertial navigationdata, and detecting at least one target road traffic marking in theground region image and inversing three-dimensional coordinates of thetarget road traffic marking, so as to reach a purpose of improvingaccuracy of a detecting result and practicability of data processing.

Based on the foregoing principle, the embodiments of this applicationprovide a data processing method. As shown in FIG. 1, the methodincludes the following step S101-step S104.

S101: Collect road data, the road data including a two-dimensionalstreetscape image, a three-dimensional point cloud, and inertialnavigation data.

The two-dimensional streetscape image may be obtained by using cameraequipment such as cameras and camera lenses to shoot a road while thevehicle is in motion, approaching an intersection, and/or when thevehicle is stopped at a stop light or stop sign. The two-dimensionalstreetscape image usually includes an image portion of a ground regionand an image portion of a non-ground region, where the image of theground region may describe data such as road surface conditions, roadtraffic markings, running vehicles on the road, and isolation strips ofthe road. The image of the non-ground region may describe buildings,plants around the road, and so on. Furthermore, the road trafficmarkings refer to signs such as lines, words, arrows, object markings,road studs, and delineators, used to transmit traffic information ofguidance, limitations, warnings, and so on. The road traffic markingsinclude, but are not limited to: lane lines and/or road signs; where theroad signs include, but are not limited to: a steering sign, adeceleration sign, a forbidden zone sign, and so on. Thethree-dimensional point cloud may be obtained by means of a laserscanner. The laser scanner may scan multiple space points in a scanningrange to obtain three-dimensional coordinates of each point in space inthe field of view of the camera, and form the three-dimensional pointcloud with reference to laser pulse strength corresponding to each spacepoint. The inertial navigation data may be obtained by using an inertialnavigation system (Inertial Navigation System, INS) and an inertialmeasurement unit (Inertial Measurement Unit, IMU) to scan. Specifically,the INS may obtain global positioning system (Global Positioning System,GPS) data of a scanning position of the laser scanner, and carrierrunning data including a speed, an acceleration, and so on of thevehicle carrying the camera, laser scanner, and the INS; the IMU mayobtain rolling, pitching, and yawing data of the carrier (e.g., thevehicle in motion).

S102: Perform region segmentation on the two-dimensional streetscapeimage and extract a ground region image based on a spatial positionrelation of the two-dimensional streetscape image, the three-dimensionalpoint cloud, and the inertial navigation data.

In the embodiments of this application, the two-dimensional streetscapeimage, the three-dimensional point cloud, and the inertial navigationdata that are collected all come from a same scene space and at the sametime. Therefore, the region segmentation may be performed on thetwo-dimensional streetscape image based on the spatial position relationof the three and by using the three-dimensional point cloud. In aspecific implementation, when the method performs step S102, the methodspecifically performs the following steps s11-s13:

s11: Establish a coordinate system of an inertial navigation of athree-dimensional space according to the inertial navigation data.

INS is an automatic navigation system that does not depend on externalinformation or radiate energy to the outside. IMU is an apparatus usedto measure a three-axis attitude angle (or an angular rate) and anacceleration of an object. The INS and the IMU both include a gyroscopeand an accelerometer, where the gyroscope forms a coordinate system ofthe inertial navigation, making a measuring axis of the accelerometersteadily form a course and an attitude angle in the coordinate system ofthe inertial navigation. The accelerometer is used to measure theacceleration when a carrier runs, and obtain the inertial navigationdata of the carrier, such as a speed, a displacement, rolling, andpitching by using a series of calculations and transformations.Therefore, the coordinate system of the inertial navigation used in thethree-dimensional space may be inversely calculated according tocollected inertial navigation data, the coordinate system of theinertial navigation being position reference of the three-dimensionalspace in a scanning process.

s12: Register the three-dimensional point cloud and the two-dimensionalstreetscape image in the three-dimensional space based on the coordinatesystem of the inertial navigation;

Regarding the coordinate system of the inertial navigation as positionreference, registering the three-dimensional point cloud and thetwo-dimensional streetscape image in the three-dimensional space. In aspecific implementation, a certain three-dimensional point P(x_(i),y_(i), z_(i)) in a scene space is obtained by using a laser scanner,where the three-dimensional point P(x_(i), y_(i), z_(i)) may obtain acertain two-dimensional point P′(x_(j)y_(j)) in the two-dimensionalstreetscape image after imaging by camera equipment. A purpose ofregistering is to find all matched three-dimensional points P(x_(i),y_(i), z_(i)) and two-dimensional points P′(x_(j), y_(j)) from thethree-dimensional point cloud and the two-dimensional streetscape image.

s13: Perform, by a registered three-dimensional point cloud, the regionsegmentation on the two-dimensional streetscape image and extract theground region image.

After registering the three-dimensional point cloud and thetwo-dimensional streetscape image in the three-dimensional space, theregion segmentation on the two-dimensional streetscape image areconveniently implemented by using the three-dimensional point cloud.Each point in the three-dimensional point cloud is represented bythree-dimensional coordinates (x, y, z), so the region segmentation maybe intuitively performed to the three-dimensional space according tovalues of x, y, and z. For example, if a value of a z axis of thethree-dimensional point in a ground region is zero, a spatial regionconstructed with all three-dimensional points whose values of z axes arezero in the three-dimensional point cloud is a ground spatial region;similarly, if a value of a z axis of the three-dimensional point in atarget spatial region is a preset value, a spatial region constructed byextracting all three-dimensional points whose values of z axes are thepreset value in the three-dimensional point cloud is the target spatialregion. Based on this, the step may use the three-dimensionalcoordinates of each point in the three-dimensional point cloud toperform the region segmentation on the three-dimensional space, and findall two-dimensional points registering with each three-dimensional pointin the ground spatial region. A region image constructed with thetwo-dimensional points in the two-dimensional streetscape image is theground region image.

Aiming at the foregoing steps s11-s13 and with reference to FIG. 2a-FIG. 2 c, FIG. 2a shows collected two-dimensional streetscape image. Instep s12, after registering the two-dimensional streetscape image shownin FIG. 2a with the three-dimensional point cloud based on thecoordinate system of the inertial navigation, an image shown in FIG. 2amay be formed. The ground region image shown in FIG. 2c may be obtainedafter performing the region segmentation, as in s13, to an image shownin FIG. 2 b.

S103: Detect at least one target road traffic marking in the groundregion image, the road traffic marking including a lane line and/or aroad sign.

During shooting the two-dimensional streetscape image, a shooting mannerof the ground region image is usually non-orthographic: camera equipmentand a road surface are basically in a head-up range, that is, an imagingaxis of the camera equipment is not vertical to the ground, but has acertain angle, and the angle in the coordinate system of the inertialnavigation represents a yaw angle and/or a pitch angle of the cameraequipment. The depth of field of the ground region image obtained in anon-orthographic manner is relatively far with gradual transition fromthe near to the distant, which provides a basis for extracting the roadtraffic marking. However, the non-orthographic manner may deform theroad traffic marking in the ground region image, increasing difficultyof detection. Based on this, the step needs to process the ground regionimage during detection, when the method performs step S103, the methodspecifically performs the following steps s21-s23:

s21: Perform binary processing to the ground region image to obtain abinary ground region image (e.g., as shown in FIG. 2d ).

Generally, the collected two-dimensional streetscape image is a colorfulimage, and the ground region image extracted from the two-dimensionalstreetscape image is also a colorful image, while the road trafficmarking is white. To make the road traffic marking in the ground regionimage more clear and more easy to detect, the binary processing needs tobe performed to the colorful ground region image in the step, so as toobtain a binary ground region image.

s22: Perform orthographic projection transformation to the binary groundregion image to generate a ground orthographic projection image (e.g.,as shown in FIG. 3e ).

A purpose of the orthographic projection transformation is to adjust adirection of an optical axis of the ground region image to be vertical;specifically, the binary ground region image may be projected to aground spatial region described by the three-dimensional point cloud togenerate the ground orthographic projection image.

s23: Extract at least one road traffic marking from the groundorthographic projection image.

The ground orthographic projection image clearly presents road trafficmarkings, such as lane lines and road signs, so that the at least oneroad traffic marking may be recognized and extracted by using imagereorganization; in a specific implementation, a method of Houghtransform (Hough Transform) may be used to recognize the lane lines inthe ground orthographic projection image and a method of module matchingmay be used to recognize the road signs in the ground orthographicprojection image.

Aiming at the foregoing steps s21-s23 and with reference to FIG. 2d-FIG. 2 e, the binary ground region image shown in FIG. 2d may beobtained aimed at the extracted ground region image shown in FIG. 2cafter the binary processing in step s21. The ground orthographicprojection shown in FIG. 2e may be obtained after the orthographicprojection transformation is performed in step s21 to the binary groundregion image shown in FIG. 2 d. The at least one road traffic markingmay be extracted from the ground orthographic projection shown in FIG.2d by using the image reorganization.

In practical application, there are always interferences, such asrunning vehicles, isolation belts, and road stains in the ground regionimage. Therefore, interference images exist in the binary groundorthographic projection image, and the interference images may affect adetecting result of the road traffic marking. As shown in FIG.3(a)-3(b), there are two lane dividers that define the lane that thevehicle is in, however, the binary ground image includes pixels thatcorrespond to the center divider fence, and making the lane divider nearthe center divider fence difficult to recognize. To resolve the problem,the method according to the embodiments of this application specificallyperforms the following steps s231-s234 when the method performs steps23:

s231: Extract at least one first road traffic marking from the groundorthographic projection image and extract an interference image by usingan image recognition method (e.g., any object recognition method basedon machine vision, neural network, statistical method, binarization andthresholding method, etc.). The original image shown in FIG. 3(c) isprocessed in accordance with the above method to obtain the groundorthographic projection image, and to obtain at least one first roadtraffic marking (e.g., lane divider on the right side portion of theimage) from the ground orthographic projection image. The road trafficmarking close to the center divider fence could not be extracted clearlydue to the presence of the center divider fence, as shown in the imageshown in FIG. 3(b). An interference image shown in FIG. 3(d) isextracted from the original image 3(c), e.g., using an object/imagerecognition method. The method can be the same as that used to obtainthe first road traffic marking, but with different parameters. Theinterference image includes pixels that do not correspond to roadtraffic marking (e.g., pixels that correspond to the metal fence centerdivider).

s232: Regard the interference image as a binary mask to re-act on theground orthographic projection image. For example, the image in FIG.3(d) is used as a mask for the image in FIG. 3(b), such that the pixelscorresponding to the traffic marking near the center divider fenceremain in the image, while the pixels that correspond to the centerdivider fence and other interference objects are removed from the image.FIG. 3B shows that, the re-acted ground image with the pixels of thecenter divider fence removed.

s233: Extract at least one second road traffic marking from a re-actedground orthographic projection image by using the image recognitionmethod. For example, the road marking next to the center divider isrecognized based on the pixels that remain in the re-acted groundorthographic projection image which correspond to the traffic markingnear the center divider fence remain in the image.

s234: Combine the first road traffic marking and the second road trafficmarking that are extracted to obtain at least one target road trafficmarking. As shown in FIG. 3(e), the both lane dividers are recognized inthe image.

Step s231-step s234 show that influence of the interference may beremoved and accuracy of the detecting result of the road traffic markingmay be improved by secondly extracting the road traffic marking;specifically, first extracting the road traffic marking for once fromthe ground orthographic projection image, at the same time extracting aninterference image as a binary mask to react on the ground orthographicprojection image; the interference image in the ground orthographicprojection image may be removed by reaction, and extracting the roadtraffic marking for another time from the ground orthographic projectionimage to combine two road traffic markings. It should be noted that toremove the influence of the interference, a second extracting manner isnot limited, for example, a third or more times extracting may be usedto combine multi-extracted road traffic markings. Compared with thesecond extracting, the third or more times extracting may remove theinfluence of the interference more thoroughly at a cost of lowefficiency, so selection in a specific implementation may be madeaccording to actual needs.

S104: Calculate three-dimensional coordinates of the at least one targetroad traffic marking based on the spatial position relation of thetwo-dimensional streetscape image, the three-dimensional point cloud,and the inertial navigation data.

In a specific implementation, when the method performs step S104, themethod specifically performs the following steps s31-s32:

s31: Obtain morphological data of the ground orthographic projectionimage in the coordinate system of the inertial navigation, where themorphological data includes position data and posture data.

s32: Find the three-dimensional coordinates of the at least one targetroad traffic marking according to the morphological data.

In the foregoing steps s31-s32, position data and posture data of theground orthographic projection image in the coordinate system of theinertial navigation may be obtained based on the spatial positionrelation of the two-dimensional streetscape image, the three-dimensionalpoint cloud, and the inertial navigation data because the groundorthographic projection image comes from the two-dimensional streetscapeimage. The three-dimensional coordinates of the road traffic marking maybe inversed by using the position data and the posture data of theground orthographic projection image in the coordinate system of theinertial navigation.

The embodiments of this application provide a data processing method.After collecting a two-dimensional streetscape image, athree-dimensional point cloud, and inertial navigation data, performingregion segmentation on the two-dimensional streetscape image andextracting a ground region image by using the three-dimensional pointcloud and the inertial navigation data, and detecting at least onetarget road traffic marking in the ground region image and inversingthree-dimensional coordinates of the target road traffic marking. It maybe avoided that a detecting result is affected if a point cloud of thethree-dimensional point cloud is sparse, occluded, missing, orsomething, because the road traffic marking is detected and obtainedfrom the two-dimensional streetscape image rather than thethree-dimensional point cloud. In addition, by accurately performingregion segmentation on, detecting, and extracting the two-dimensionalstreetscape image by using the inertial navigation data and thethree-dimensional point cloud, accuracy of the detecting result andpracticability of data processing may be effectively improved.

As disclosed herein, the advanced driver assistance system (ADAS) can beinstalled on a vehicle, and utilize various types of sensors availableon the vehicle to collect the surrounding environment data, and identifyobjects in the surroundings, tracking and following moving objects inthe surroundings, and alert the driver of potential danger on the road,and/or automatically adjust control of the vehicle to avoid danger onthe road.

The vehicle moves on a street, the vehicle collects 2 dimensional imagesof the streetscape using a camera (e.g., a camera installed on the frontof the vehicle and/or on the top of the vehicle). In addition, thevehicle collects point cloud data using specialized equipment whichincludes light sensors and laser scanners. The laser scanner scans thespace in front of the vehicle using laser beams, and collect reflectiondata which reflects the distance and shapes of objects in front of thevehicle (e.g., ground, road surface, other vehicles, buildings, roadsigns, reflective road markings, etc.). The two dimensional streetscapedata and the three-dimensional point cloud data are taking concurrently,and can be correlated with one another. The point cloud data is used tosegment out the road surface from the two-dimensional streetscape image.The pixel data in the two-dimensional streetscape image are projectedinto the three-dimensional point cloud of the space in front of thevehicle. Specifically, the road surface image in the two-dimensionalstreetscape image is transformed into an orthographic projection imageand road and traffic markings are extracted from the orthographicprojection image based on the features extracted from orthographicprojection image. In the end, based on the position of the road andtraffic markings in the orthographic image, the actual three-dimensionalcoordinates of the road and traffic markings are derived in athree-dimensional coordinate system of the vehicle.

FIG. 4 is an exemplary process flow for getting the traffic markings andremoving interferences described above. In the process flow, the processstarts (402) with registering (404) the three dimensional point clouddata with the two-dimensional streetscape image taken at the same time.Then, segmentation (406) of the road traffic markings is performed. Theimage is binarized (408). The image is projected to the ground surfaceand orthographic projection of the road surface is obtained (410). Theroad marking is extracted from the orthographic projection of the roadsurface (412). If there are portions of the image that is unrecognized(414), an interference image is generated and a binarized mask iscreated based on the interference image (422), and the interferenceimage is used as a mask to process the image to remove the interferingpixels. The processed image undergoes the process to identify roadmarking again (420). In the end, when all markings are identified, andthere are no more unrecognized portions in the image, the recognizedroad markings are combined (416). The process ends (418).

FIG. 5 shows an exemplary process for extracting a road marking from theorthographic projection image. The process begins (502). Regionsegmentation is performed (504). If segmentation is successful (506),skeleton of the road markings are extracted (508), otherwise, theprocess ends (518). After the skeleton of the road markings areextracted, center line detection is performed (510). Orthographicprojection distance map is generated (512). Line detection (514) andpattern recognition based on templates of road markings (516) areperformed. The process restarts when additional images are acquired.

Embodiments of this application provide a terminal, including devicessuch as laptops, phones, pads (PAD), and vehicle terminals. Referring toFIG. 6, an internal structure of the terminal may include, but is notlimited to: a processor, a user interface, a network interface, and amemory. The processor, the user interface, the network interface, andthe memory in the terminal may be connected with a bus or in other way.In FIG. 6 of the embodiments of this application, take a bus connectionas an example.

The user interface is an interaction and information exchange mediumbetween a user and a terminal, specifically including a display(Display) used to output, a keyboard (Keyboard) used to input, and soon. It should be noted that the keyboard here may be a physicalkeyboard, or a touch-screen virtual keyboard, or a physical andtouch-screen virtual keyboard. The processor (or central processing unit(CPU)) is a calculation core and a control core of the terminal, whichcan analyze instructions in the terminal and data of a processingterminal, for example, the CPU may be used to analyze an on/offinstruction sent by the user to the terminal, and control the terminalperform on/off operations; for another example, the CPU may transmitinteraction data between internal structures of the terminal, and thelike. The memory (Memory) is a memory device in the terminal, used forstoring programs and data. It should be understood that the memory heremay include an internal memory of the terminal, and may also include anexpanded memory supported by the terminal. The memory provides a storagespace, storing an operating system of the terminal. In the embodimentsof this application, the storage space of the memory also stores a dataprocessing apparatus. The terminal performs corresponding steps of amethod procedure shown in the foregoing FIGS. 1-5 by sunning the dataprocessing apparatus in the memory. Referring to FIG. 7, the dataprocessing apparatus performs the following units:

-   -   a collection unit 101, configured to collect road data, the road        data including a two-dimensional streetscape, a        three-dimensional point cloud, and inertial navigation data.    -   a segmentation processing unit 102, configured to perform region        segmentation on the two-dimensional streetscape image and        extract a ground region image based on a spatial position        relation of the two-dimensional streetscape image, the        three-dimensional point cloud, and the inertial navigation data.    -   a detection unit 103, configured to detect at least one target        road traffic marking in the ground region image, the road        traffic marking including a lane line and/or a road sign.    -   a calculation unit 104, configured to calculate        three-dimensional coordinates of the at least one target road        traffic marking based on the spatial position relation of the        two-dimensional streetscape image, the three-dimensional point        cloud, and the inertial navigation data.

In a specific implementation, when the device operates the segmentationprocessing unit 102, the device specifically operates the followingunits:

-   -   an establishing unit 1001, configured to establish a coordinate        system of an inertial navigation of a three-dimensional space        according to the inertial navigation data.    -   an registering unit 1002, configured to register the        three-dimensional point cloud and the two-dimensional        streetscape image in the three-dimensional space based on the        coordinate system of the inertial navigation.    -   a region segmentation unit 1003, configured to perform, by a        registered three-dimensional point cloud, the region        segmentation on the two-dimensional streetscape image.    -   an image extracting unit 1004, configured to extract the ground        region image.

In a specific implementation, when the device operates the detectionunit 103, the device specifically operates the following units:

-   -   a binary processing unit 2001, configured to perform binary        processing to the ground region image to obtain a binary ground        region image.    -   a transformation unit 2002, configured to perform orthographic        projection transformation to the binary ground region image to        generate a ground orthographic projection image.    -   a marking extracting unit 2003, configured to extract at least        one road traffic marking from the ground orthographic projection        image.

In a specific implementation, when the device operates the markingextracting unit 2003, the device specifically operates the followingunits:

-   -   a first extracting unit 2331, configured to extract at least one        first road traffic marking from the ground orthographic        projection image and extract an interference image by using an        image recognition method.    -   a reaction processing unit 2332, configured to regard the        interference image as a binary mask to react on the ground        orthographic projection image.    -   a second extracting unit 2333, configured to extract at least        one second road traffic marking from a reacted ground        orthographic projection image by using the image recognition        method.    -   a combination processing unit 2334, configured to combine the        first road traffic marking and the second road traffic marking        that are extracted to obtain at least one target road traffic        marking.

In a specific implementation, when the device operates the calculationunit 104, the device specifically operates the following units:

-   -   a morphological data obtaining unit 3001, configured to obtain        morphological data of the ground orthographic projection image        in the coordinate system of the inertial navigation, where the        morphological data includes position data and posture data.    -   a coordinate finding unit 3002, configured to find the        three-dimensional coordinates of the at least one target road        traffic marking according to the morphological data.

The data processing method shown in FIGS. 1-5 may be performed by everyunit in the data processing apparatus shown in FIG. 7 according to anembodiment of this application. For example, step S101, S102, S103, andS104 shown in FIG. 1 may be separately performed by the collection unit101, the segmentation processing unit 102, the detection unit 103, andcalculation unit 104 shown in FIG. 7; step s11, s12, and s13 may beseparately performed by the establishing unit 1001, the registering unit1002, the region segmentation unit 1003, and the image extracting unit1004 shown in FIG. 7; step s21, s22, and s23 may be separately performedby the binary processing unit 2001, the transformation unit 2002, andthe marking extracting unit 2003 shown in FIG. 7; step s231, s232, s233,and s234 may be separately performed by the first extracting unit 2331,the reaction processing unit 2332, the second extracting unit 2333, andthe combination processing unit 2334 shown in FIG. 7; step s31 and s32may be separately performed by the morphological data obtaining unit3001 and the coordinate finding unit 3002 shown in FIG. 7.

According to another embodiment of this application, units of the systemfor data processing apparatus shown in FIG. 7 may be separately orwholly combined into one or several other units, or one (or more) of theunits herein may further be divided into multiple units of smallerfunctions. In this way, same operations can be implemented, andimplementation of the technical effects of the embodiments of thisapplication is not affected. The foregoing units are divided based onlogical functions. In an actual application, a function of one unit mayalso be implemented by multiple units, or functions of multiple unitsare implemented by one unit. In other embodiments of this application,the terminal device may include other modules. However, in an actualapplication, these functions may also be cooperatively implemented byanother unit and may be cooperatively implemented by multiple units.

According to another embodiment of this application, a data processingapparatus shown in FIG. 7 may be constructed and a data processingmethod may be implemented according to the embodiments of thisapplication by running a computer program (including program code) thatcan perform the data processing method shown in FIG. 1 on processingelements and memory elements including a central processing unit (CPU),a random access memory (RAM), a read-only memory (ROM), and so on, forexample, generic computing devices of computers. The computer programmay be recorded in, for example, a computer readable recording medium,and may be loaded into the foregoing computing device by using thecomputer readable recording medium, and run in the computing device.

Being same with the method, the embodiments of this application providea data processing apparatus. After collecting a two-dimensionalstreetscape image, a three-dimensional point cloud, and inertialnavigation data, performing region segmentation on the two-dimensionalstreetscape image and extracting a ground region image by using thethree-dimensional point cloud and the inertial navigation data, anddetecting at least one target road traffic marking in the ground regionimage and inversing three-dimensional coordinates of the target roadtraffic marking. It may be avoided that a detecting result is affectedif a point cloud of the three-dimensional point cloud is sparse,occluded, missing, or something, because the road traffic marking isdetected and obtained from the two-dimensional streetscape image ratherthan the three-dimensional point cloud. In addition, by accuratelyperforming region segmentation on, detecting, and extracting thetwo-dimensional streetscape image by using the inertial navigation dataand the three-dimensional point cloud, accuracy of the detecting resultand practicability of data processing may be effectively improved.

It should be understood that each part of this application may beimplemented by using hardware, software, firmware, or their combination.In the foregoing implementations, multiple steps or methods may beimplemented by using the software or firmware that are stored in amemory and executed by an appropriate instruction execution system. Forexample, if using the hardware to implement, as same as in anotherimplementation, any one of the following technologies known in the fieldor their combination may be used to implement: a discrete logicalcircuit of a logical gating circuit used to implement a logic functionfor a digital signal, an appropriate specified integrated circuit of acombinational logical gating circuit, a programmable gate array (PGA), afield programmable gate array (FPGA), and so on.

A person of ordinary skill in the art may understand that all or some ofthe processes of the methods in the embodiments may be implemented by acomputer program instructing relevant hardware. The program may bestored in a computer readable storage medium. When the program runs, theprocesses of the methods in the embodiments are performed. In addition,functional units in the embodiments of the present application may beintegrated into one processing module, or each of the units may existalone physically, or two or more units are integrated into one module.The integrated modules may be implemented in the form of hardware orsoftware functional modules. If implemented in the form of softwarefunctional modules and sold or used as an independent product, theintegrated modules may also be stored in a computer-readable storagemedium. The storage medium may be a magnetic disk, an optical disc, aread-only memory (Read-Only Memory, ROM) or a random access memory(Random Access Memory, RAM), or the like.

What is disclosed above is merely exemplary embodiments of thisapplication, and certainly is not intended to limit the protection scopeof this application. Therefore, equivalent variations made in accordancewith the claims of this application shall fall within the scope of thisapplication.

What is claimed is:
 1. A method, comprising: at a computing systemcomprising one or more processors and memory, the computing system isattached to a moving vehicle traveling along a road: collecting roaddata while the computing system is in motion, the road data comprising atwo-dimensional streetscape image, a three-dimensional point cloud, andinertial navigation data; performing region segmentation on thetwo-dimensional streetscape image; extracting a ground region imagebased on a spatial position relation of the two-dimensional streetscapeimage, the three-dimensional point cloud, and the inertial navigationdata; detecting at least one target road traffic marking in the groundregion image; and controlling motion of the moving vehicle in accordancewith three-dimensional coordinates of the at least one target roadtraffic marking that are calculated based on the spatial positionrelation of the two-dimensional streetscape image, the three-dimensionalpoint cloud, and the inertial navigation data.
 2. The method accordingto claim 1, wherein performing region segmentation on thetwo-dimensional streetscape image comprises: establishing a coordinatesystem of an inertial navigation of a three-dimensional space accordingto the inertial navigation data; registering the three-dimensional pointcloud and the two-dimensional streetscape image in the three-dimensionalspace based on the coordinate system of the inertial navigation;performing, by a registered three-dimensional point cloud, the regionsegmentation on the two-dimensional streetscape image.
 3. The methodaccording to claim 1, wherein the detecting at least one target roadtraffic marking in the ground region image comprises: performing binaryprocessing to the ground region image to obtain a binary ground regionimage; performing orthographic projection transformation to the binaryground region image to generate a ground orthographic projection image;and extracting the at least one road traffic marking from the groundorthographic projection image.
 4. The method according to claim 3,wherein extracting the at least one road traffic marking from the groundorthographic projection image comprises: extracting at least one firstroad traffic marking from the ground orthographic projection image;extracting an interference image from the ground orthographic projectionimage by using an image recognition method; using the interference imageas a binary mask to filter the ground orthographic projection image;extracting at least one second road traffic marking from the filteredground orthographic projection image; and combining the first roadtraffic marking and the second road traffic marking that are extractedto obtain the at least one target road traffic marking.
 5. The methodaccording to claim 3, wherein calculating three-dimensional coordinatesof the at least one target road traffic marking based on the spatialposition relation of the two-dimensional streetscape image, thethree-dimensional point cloud, and the inertial navigation datacomprises: obtaining morphological data of the ground orthographicprojection image in the coordinate system of the inertial navigation,wherein the morphological data comprises position data and posture data;and finding the three-dimensional coordinates of the at least one targetroad traffic marking according to the morphological data.
 6. Anon-transitory computer-readable storage medium storing instructions,the instructions, when executed by one or more processors of at acomputing system that is attached to a moving vehicle traveling along aroad, cause the processors to perform operations comprising: collectingroad data while the computing system is in motion, the road datacomprising a two-dimensional streetscape image, a three-dimensionalpoint cloud, and inertial navigation data; performing regionsegmentation on the two-dimensional streetscape image; extracting aground region image based on a spatial position relation of thetwo-dimensional streetscape image, the three-dimensional point cloud,and the inertial navigation data; detecting at least one target roadtraffic marking in the ground region image; and controlling motion ofthe moving vehicle in accordance with three-dimensional coordinates ofthe at least one target road traffic marking that are calculated basedon the spatial position relation of the two-dimensional streetscapeimage, the three-dimensional point cloud, and the inertial navigationdata.
 7. The computer-readable storage medium according to claim 6,wherein performing region segmentation on the two-dimensionalstreetscape image comprises: establishing a coordinate system of aninertial navigation of a three-dimensional space according to theinertial navigation data; registering the three-dimensional point cloudand the two-dimensional streetscape image in the three-dimensional spacebased on the coordinate system of the inertial navigation; performing,by a registered three-dimensional point cloud, the region segmentationon the two-dimensional streetscape image.
 8. The computer-readablestorage medium according to claim 6, wherein the detecting at least onetarget road traffic marking in the ground region image comprises:performing binary processing to the ground region image to obtain abinary ground region image; performing orthographic projectiontransformation to the binary ground region image to generate a groundorthographic projection image; and extracting the at least one roadtraffic marking from the ground orthographic projection image.
 9. Thecomputer-readable storage medium according to claim 8, whereinextracting the at least one road traffic marking from the groundorthographic projection image comprises: extracting at least one firstroad traffic marking from the ground orthographic projection image;extracting an interference image from the ground orthographic projectionimage by using an image recognition method; using the interference imageas a binary mask to filter the ground orthographic projection image;extracting at least one second road traffic marking from the filteredground orthographic projection image; and combining the first roadtraffic marking and the second road traffic marking that are extractedto obtain the at least one target road traffic marking.
 10. Thecomputer-readable storage medium according to claim 8, whereincalculating three-dimensional coordinates of the at least one targetroad traffic marking based on the spatial position relation of thetwo-dimensional streetscape image, the three-dimensional point cloud,and the inertial navigation data comprises: obtaining morphological dataof the ground orthographic projection image in the coordinate system ofthe inertial navigation, wherein the morphological data comprisesposition data and posture data; and finding the three-dimensionalcoordinates of the at least one target road traffic marking according tothe morphological data.
 11. A computing system, comprising: one or moreprocessors; and memory storing instructions, wherein: the computingsystem is attached to a moving vehicle traveling along a road, and theinstructions, when executed by the one or more processors, cause theprocessors to perform operations comprising: collecting road data whilethe computing system is in motion, the road data comprising atwo-dimensional streetscape image, a three-dimensional point cloud, andinertial navigation data; performing region segmentation on thetwo-dimensional streetscape image; extracting a ground region imagebased on a spatial position relation of the two-dimensional streetscapeimage, the three-dimensional point cloud, and the inertial navigationdata; detecting at least one target road traffic marking in the groundregion image; and controlling motion of the moving vehicle in accordancewith three-dimensional coordinates of the at least one target roadtraffic marking that are calculated based on the spatial positionrelation of the two-dimensional streetscape image, the three-dimensionalpoint cloud, and the inertial navigation data.
 12. The computing systemaccording to claim 11, wherein performing region segmentation on thetwo-dimensional streetscape image comprises: establishing a coordinatesystem of an inertial navigation of a three-dimensional space accordingto the inertial navigation data; registering the three-dimensional pointcloud and the two-dimensional streetscape image in the three-dimensionalspace based on the coordinate system of the inertial navigation;performing, by a registered three-dimensional point cloud, the regionsegmentation on the two-dimensional streetscape image.
 13. The computingsystem according to claim 11, wherein the detecting at least one targetroad traffic marking in the ground region image comprises: performingbinary processing to the ground region image to obtain a binary groundregion image; performing orthographic projection transformation to thebinary ground region image to generate a ground orthographic projectionimage; and extracting the at least one road traffic marking from theground orthographic projection image.
 14. The computing system accordingto claim 13, wherein extracting the at least one road traffic markingfrom the ground orthographic projection image comprises: extracting atleast one first road traffic marking from the ground orthographicprojection image; extracting an interference image from the groundorthographic projection image by using an image recognition method;using the interference image as a binary mask to filter the groundorthographic projection image; extracting at least one second roadtraffic marking from the filtered ground orthographic projection image;and combining the first road traffic marking and the second road trafficmarking that are extracted to obtain the at least one target roadtraffic marking.
 15. The computing system according to claim 13, whereincalculating three-dimensional coordinates of the at least one targetroad traffic marking based on the spatial position relation of thetwo-dimensional streetscape image, the three-dimensional point cloud,and the inertial navigation data comprises: obtaining morphological dataof the ground orthographic projection image in the coordinate system ofthe inertial navigation, wherein the morphological data comprisesposition data and posture data; and finding the three-dimensionalcoordinates of the at least one target road traffic marking according tothe morphological data.